The Open Inference Project
Making the analytic context of evidence visible
The conjecture
Scientific results do not arrive without context. Their meaning depends on the questions asked, the populations and outcomes examined, the models that were fit, and the alternative analytic paths that were available.
This Research Forum presentation began with a simple conjecture: the analytic context of evidence is systematically underappreciated. Published results are highly visible, while much of the analytic process that produced them can be difficult for readers to reconstruct.
Central Idea
The problem is not analytic complexity. The problem is analytic complexity that readers cannot see.
Where analytic search space comes from
Multiple outcomes
Primary, secondary, exploratory, and time-dependent outcomes create multiple reasonable questions within the same study.
Multiple models
Covariate sets, functional forms, model specifications, and variable definitions can each change the analysis.
Multiple subgroups
Age, disease severity, biomarkers, and other patient characteristics create additional analytic pathways.
Multiple decisions
Inclusion criteria, missing-data handling, transformations, and other defensible choices expand the space of possible analyses.
Studying the analytic structure of a literature
Rather than treating analytic flexibility and multiplicity as purely theoretical concerns, the project asked an empirical question: how large is the analytic search space in contemporary dermatology research?
An initial manual review suggested a substantial analytic search space. The work was then expanded into a structured, reproducible pipeline designed to characterize statistical reporting patterns across a much larger corpus of articles from the Journal of the American Academy of Dermatology.
Early Signal
Across the JAAD corpus, statistical testing is highly visible, while multiplicity adjustment, preregistration, and alternative inferential frameworks appear far less often.
What the review suggests
- Researchers often have many reasonable ways to analyze the same data.
- P-values are extremely common in the published dermatology literature.
- Adjustments for multiple hypothesis testing are comparatively uncommon.
- Preregistration is rare outside clinical trials.
- Bayesian approaches are rarely used.
- Analytic flexibility frequently remains implicit rather than directly visible to the reader.
What can we do about it?
Investigator level
Improve how individual studies are designed, analyzed, interpreted, and shared. Define questions clearly, distinguish confirmation from exploration, prespecify consequential choices when feasible, and communicate uncertainty.
Systems level
Move beyond transparency within one paper and make the analytic structure of an entire evidence base easier to discover, examine, compare, and interpret.
A Two-Level Response
Transparent individual studies are necessary, but they do not by themselves make the structure of a literature visible.
The Open Inference Project
The Open Inference Project is a research program focused on the structure behind published evidence. It shifts attention from an isolated reported result to the larger analytic context that produced it.
The goal is to make features such as analytic flexibility, multiplicity, preregistration, robustness, and interpretability more visible across the scientific literature—and ultimately to help readers understand not only what a study found, but the analytic context in which that result should be interpreted.
The Open Inference Project
Better inference requires making the analytic context of evidence visible.
Why bring this to the Research Forum?
The presentation was intentionally framed as work in progress. The empirical analysis, measurement strategy, and broader Open Inference framework were presented while they were still open to criticism and revision.
That reflects the purpose of the SoCO Research Forum itself: bringing ideas forward early enough that colleagues can challenge assumptions, expose weaknesses, and improve the work before the story becomes fixed.
Society of Cutaneous Oncology Research Forum. Research discussions presented through the Forum may include work in progress and should be interpreted in that context.